System and method for multiple viewing-window display of capsule images

ABSTRACT

Systems and methods are provided for displaying images captured from a capsule camera system. In order to increase the efficiency of viewing the image sequence, the image sequence is divided into multiple sub-sequences and the multiple sub-sequences are displayed in multiple viewing windows on a display screen concurrently. For images from a panoramic capsule system, the images typically have a very wide aspect ratio and may require different configuration for displaying in multiple viewing windows than that for image sequence having non-wide aspect ratio. The present invention also discloses methods and systems that divide a sequence from panoramic capsule camera into multiple member sequences and form an aggregated video. The aggregated video not only makes viewing more comfortable, but also speeds up viewing time.

CROSS REFERENCE TO RELATED APPLICATIONS

The present invention is related and claims priority to U.S. patentapplication Ser. No. 12/577,626, entitled “SYSTEM AND METHOD FOR DISPLAYOF PANORAMIC CAPSULE IMAGES” and filed on Oct. 12, 2009. The U.S. patentapplication is hereby incorporated by reference in its entireties.

FIELD OF THE INVENTION

The present invention relates to diagnostic imaging inside the humanbody. In particular, the present invention relates to displaying imagescaptured by a capsule camera system using multiple viewing windows.

BACKGROUND

Devices for imaging body cavities or passages in vivo are known in theart and include endoscopes and autonomous encapsulated cameras.Endoscopes are flexible or rigid tubes that pass into the body throughan orifice or surgical opening, typically into the esophagus via themouth or into the colon via the rectum. An image is formed at the distalend using a lens and transmitted to the proximal end, outside the body,either by a lens-relay system or by a coherent fiber-optic bundle. Aconceptually similar instrument might record an image electronically atthe distal end, for example using a CCD or CMOS array, and transfer theimage data as an electrical signal to the proximal end through a cable.Endoscopes allow a physician control over the field of view and arewell-accepted diagnostic tools. However, they do have a number oflimitations, present risks to the patient, are invasive anduncomfortable for the patient, and their cost restricts theirapplication as routine health-screening tools.

Because of the difficulty traversing a convoluted passage, endoscopescannot reach the majority of the small intestine and special techniquesand precautions, that add cost, are required to reach the entirety ofthe colon. Endoscopic risks include the possible perforation of thebodily organs traversed and complications arising from anesthesia.Moreover, a trade-off must be made between patient pain during theprocedure and the health risks and post-procedural down time associatedwith anesthesia. Endoscopies are necessarily inpatient services thatinvolve a significant amount of time from clinicians and thus arecostly.

An alternative in vivo image sensor that addresses many of theseproblems is capsule endoscope. A camera is housed in a swallowablecapsule, along with a radio transmitter for transmitting data, primarilycomprising images recorded by the digital camera, to a base-stationreceiver or transceiver and data recorder outside the body. The capsulemay also include a radio receiver for receiving instructions or otherdata from a base-station transmitter. Instead of radio-frequencytransmission, lower-frequency electromagnetic signals may be used. Powermay be supplied inductively from an external inductor to an internalinductor within the capsule or from a battery within the capsule.

An autonomous capsule camera system with on-board data storage wasdisclosed in the U.S. patent application Ser. No. 11/533,304, entitled“In Vivo Autonomous Camera with On-Board Data Storage or DigitalWireless Transmission in Regulatory Approved Band,” filed on Sep. 19,2006. This application describes a capsule system using on-board storagesuch as semiconductor nonvolatile archival memory to store capturedimages. After the capsule passes from the body, it is retrieved. Capsulehousing is opened and the images stored are transferred to a computerworkstation for storage and analysis. For capsule images either receivedthrough wireless transmission or retrieved from on-board storage, theimages will have to be displayed and examined by diagnostician toidentify potential anomalies.

Besides the above mentioned forward-looking capsule cameras, there areother types of capsule cameras that provide side view or panoramic view.A side or reverse angle is required in order to view the tissue surfaceproperly. Conventional devices are not able to see such surfaces, sincetheir FOV is substantially forward looking. It is important for aphysician to see all areas of these organs, as polyps or otherirregularities need to be thoroughly observed for an accurate diagnosis.Since conventional capsules are unable to see the hidden areas aroundthe ridges, irregularities may be missed, and critical diagnoses ofserious medical conditions may be flawed. A camera configured to capturea panoramic image of an environment surrounding the camera is disclosedin U.S. patent application Ser. No. 11/642,275, entitled “In vivo sensorwith panoramic camera” and filed on Dec. 19, 2006. The panoramic camerais configured with a longitudinal field of view (FOV) defined by a rangeof view angles relative to a longitudinal axis of the capsule and alatitudinal field of view defined by a panoramic range of azimuth anglesabout the longitudinal axis such that the camera can capture a panoramicimage covering substantially a 360 deg latitudinal FOV.

Conceptually, multiple individual cameras may be configured to covercompletely or substantially a 360 deg latitudinal FOV. However, suchpanoramic capsule system may be expensive since multiple image sensorsand associated electronics may be required. A cost-effective panoramiccapsule system is disclosed in U.S. patent application Ser. No.11/624,209, entitled “Panoramic Imaging System”, filed on Jan. 17, 2007.The panoramic capsule system uses an optical system configured tocombine several fields-of-view to cover a 360° view. Furthermore, thecombined fields-of-view is projected onto a single sensor to save cost.Therefore, this single sensor capsule system functions effectively asmultiple cameras at a lower cost.

Similar to the situation with a conventional forward looking capsulesystem, for side-view and panoramic view capsule systems with digitalwireless transmission or on-board storage, the captured images will beplayed back for analysis and examination. During playback, thediagnostician looks to find polyps or other points of interest asquickly and efficiently as possible. The playback is at a controllableframe rate and may be increased to reduce viewing time. However, if theframe rate is increased too much, the gyrations of the field of view(FOV) will make the video stream difficult to follow. At whatever framerate, image gyration demands more cognitive effort on thediagnostician's part to follow, resulting in viewer fatigue andincreased chance of missing important information in the video.

For images associated with either a conventional capsule camera withforward-looking view, a capsule camera with side view or a panoramiccamera with a panoramic view, the images will be viewed by diagnosticianon a viewing station or a display device. Due to the large amount ofimage data to be examined and the cost associated with thediagnostician's time, it is desired that the video corresponding to theimage data can be displayed in a way that will help reduce thediagnostician viewing time without compromising the quality andreliability of the diagnostics.

SUMMARY

The present invention provides an effective method and system forviewing an image sequence generated from a capsule camera system. In oneembodiment, a method for displaying video of images from a capsulecamera system is disclosed which comprises accepting images capturedwith the capsule camera system, generating video member sequences (i.e.,sub-sequences) based on the images, composing an aggregated videocomprising a plurality of the video member sequences and providing theaggregated video. The capsule images may be captured by a single capsulecamera, or by a panoramic camera system combining multiplefields-of-view into a single panoramic image. The member sequences maybe generated by uniformly interleaving, i.e. sub-sampling the capsuleimages temporally, or by dividing the capsule images into temporallyconsecutive sections. The aggregated video, i.e., the collection of allvideo member sequences, can be composed by arranging the plurality ofvideo member sequences for displaying in multiple viewing windows of thedisplay window. In another embodiment, the method for displaying videoof panoramic images from a capsule camera system is disclosed whichcomprises further steps of pre-processing and stitching the images. Inyet another embodiment of the current invention, the method fordisplaying video of panoramic images comprising generating the membersequences by spatial shifting images of the original sequencecyclically. In another alternative embodiment of the present invention,the frame rate is provided for the aggregated video.

A system for displaying video of capsule images is also disclosed. Thesystem comprises an interface module coupled to accept capsule imagescaptured with the capsule camera system, a first processing modulecoupled to the interface module for accessing the panorama images andconfigured to generate video member sequences based on the capsuleimages, a second processing module coupled to the first processingmodule for receiving the video member sequence and configured to composean aggregated video comprising a plurality of the video membersequences, and an output interface module coupled to receive and toprovide the aggregated video. The capsule images may be captured by asingle capsule camera or by a panoramic camera system combining multiplefields-of-view into a single panoramic image. The member sequences aregenerated by uniformly interleaving the capsule images temporally ordividing the panorama images into temporally consecutive sections. Fordisplaying video of panoramic images, the member sequences can befurther generated by spatially cyclically shifting images of theoriginal sequence. The aggregated video, i.e., the collection of allvideo member sequences, can be composed by arranging the plurality ofthe video member sequences for displaying in the display window. Inanother alternative embodiment of the present invention, the frame rateis provided for the aggregated video.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows schematically a capsule camera system in the GI tract,where archival memory is used to store capsule images to be analyzedand/or examined.

FIG. 2A shows schematically a panoramic capsule camera system in the GItract, where archival memory is used to store capsule images to beanalyzed and/or examined.

FIG. 2B shows schematically a panoramic capsule camera system in the GItract, where wireless transmission is used to send panoramic images to abase station for further analysis and/or examination.

FIG. 3A shows a video screen divided into a display window fordisplaying video and information bars for displaying informationassociated with the underlying capsule images.

FIG. 3B shows an exemplary multiple viewing-windows where 8viewing-windows are used for the capsule images.

FIG. 4A shows one exemplary arrangement of display window for videomember sequences from a panoramic camera system using a reflectiveelement having 4 sides.

FIG. 4B shows an alternative exemplary arrangement of display window forvideo member sequences from a panoramic camera system using a reflectiveelement having 4 sides.

FIG. 5A shows one exemplary arrangement of display window to accommodatetwo video viewing windows simultaneously.

FIG. 5B shows an alternative exemplary arrangement of display window toaccommodate two video viewing windows simultaneously.

FIG. 5C shows another exemplary arrangement of display window toaccommodate two video viewing windows showing two images of the sameinstance where one is a spatially shifted version of the other.

FIG. 6A shows one exemplary arrangement of display window to accommodatethree video viewing windows simultaneously.

FIG. 6B shows an exemplary alternative arrangement of display window toaccommodate three video viewing windows simultaneously.

FIG. 7 shows an exemplary temporal sub-sampling method to constructmember sequences from an original sequence.

FIG. 8 shows a method of constructing member sequences from an originalsequence by equally dividing the original sequence into sections.

FIG. 9A shows an exemplary flowchart for a capsule system embodying thepresent invention.

FIG. 9B shows an exemplary flowchart for a panoramic capsule systemembodying the present invention.

DETAILED DESCRIPTION OF THE INVENTION

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,may be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the systems and methods of the present invention, asrepresented in the figures, is not intended to limit the scope of theinvention, as claimed, but is merely representative of selectedembodiments of the invention.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentmay be included in at least one embodiment of the present invention.Thus, appearances of the phrases “in one embodiment” or “in anembodiment” in various places throughout this specification are notnecessarily all referring to the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. Oneskilled in the relevant art will recognize, however, that the inventioncan be practiced without one or more of the specific details, or withother methods, components, etc. In other instances, well-knownstructures, or operations are not shown or described in detail to avoidobscuring aspects of the invention.

The illustrated embodiments of the invention will be best understood byreference to the drawings, wherein like parts are designated by likenumerals throughout. The following description is intended only by wayof example, and simply illustrates certain selected embodiments ofapparatus and methods that are consistent with the invention as claimedherein.

The present invention discloses methods and systems for multiple viewingwindows so that multiple sub-sequences of a capsule image sequence canbe displayed concurrently to increase the efficiency of examination by adiagnostician. The sub-sequence is also called a member sequence in thisdisclosure. The images may be received from a capsule camera systemhaving on-board archival memory to store the images or received from acapsule camera having wireless transmission module. FIG. 1 shows aswallowable capsule system 110 inside body lumen 100, in accordance withone embodiment of the present invention. Lumen 100 may be, for example,the colon, small intestines, the esophagus, or the stomach. Capsulesystem 110 is entirely autonomous while inside the body, with all of itselements encapsulated in a capsule housing 10 that provides a moisturebarrier, protecting the internal components from bodily fluids. Capsulehousing 10 is transparent or at least transparent over the lens and LEDareas, so as to allow light from the light-emitting diodes (LEDs) ofilluminating system 12A to pass through the wall of capsule housing 10to the lumen 100 walls, and to allow the scattered light from the lumen100 walls to be collected and imaged within the capsule camera. Capsulehousing 10 also protects lumen 100 from direct contact with foreignmaterial inside capsule housing 10. Capsule housing 10 is provided ashape that enables it to be swallowed easily and later to pass throughof the GI tract. Generally, capsule housing 10 is sterile, made ofnon-toxic material, and is sufficiently smooth to minimize the chance oflodging within the lumen.

As shown in FIG. 1, capsule system 110 includes illuminating system 12Aand a camera that includes optical system 14A and image sensor 16. Asemiconductor nonvolatile archival memory 20 may be provided to allowthe images to be stored and later retrieved at a docking station outsidethe body, after the capsule is recovered. System 110 includes batterypower supply 24 and an output port 26. Capsule system 110 may bepropelled through the GI tract by peristalsis.

Illuminating system 12A may be implemented by LEDs. In FIG. 1, the LEDsare located adjacent to the camera's aperture, although otherconfigurations are possible. The light source may also be provided, forexample, behind the aperture. Other light sources, such as laser diodes,may also be used. Alternatively, white light sources or a combination oftwo or more narrow-wavelength-band sources may also be used. White LEDsare available that may include a blue LED or a violet LED, along withphosphorescent materials that are excited by the LED light to emit lightat longer wavelengths. The portion of capsule housing 10 that allowslight to pass through may be made from bio-compatible glass or polymer.

Optical system 14A, which may include multiple refractive, diffractive,or reflective lens elements, provides an image of the lumen walls onimage sensor 16. Image sensor 16 may be provided by charged-coupleddevices (CCD) or complementary metal-oxide-semiconductor (CMOS) typedevices that convert the received light intensities into correspondingelectrical signals. Image sensor 16 may have a monochromatic response orinclude a color filter array such that a color image may be captured(e.g. using the RGB or CYM representations). The analog signals fromimage sensor 16 are preferably converted into digital form to allowprocessing in digital form. Such conversion may be accomplished using ananalog-to-digital (A/D) converter, which may be provided inside thesensor (as in the current case), or in another portion inside capsulehousing 10. The A/D unit may be provided between image sensor 16 and therest of the system. LEDs in illuminating system 12A are synchronizedwith the operations of image sensor 16. One function of control module22 is to control the LEDs during image capture operation. The controlmodule may also be responsible for other functions such as managingimage capture and coordinating image retrieval.

After the capsule camera traveled through the GI tract and exits fromthe body, the capsule camera is retrieved and the images stored in thearchival memory are read out through the output port. The receivedimages are usually transferred to a base station for processing and fora diagnostician to examine. The accuracy as well as efficiency ofdiagnostics is most important. A diagnostician is expected to examineall images and correctly identify all anomalies. In order to help thediagnostician to perform the examination more efficiently withoutcompromising the quality of examination, the received images are subjectto processing of the present invention by displaying multiplesub-sequences of the images in multiple viewing windows concurrently.The desire of using multiple viewing windows is not restricted to theconventional capsule camera. For capsule cameras having panoramic view,the need for efficient viewing for diagnostics also arises.

FIG. 2A shows an exemplary swallowable panoramic capsule system 202inside body lumen 100. Lumen 100 may be, for example, the colon, smallintestines, the esophagus, or the stomach. Panoramic capsule system 202is entirely autonomous while inside the body, with all of its elementsencapsulated in a capsule housing 10 that provides a moisture barrier,protecting the internal components from bodily fluids. Capsule housing10 is transparent or at least transparent over the areas of lens andLEDs, so as to allow light from the light-emitting diodes (LEDs) ofilluminating system 12 to pass through the wall of capsule housing 10 tothe lumen 100 walls, and to allow the scattered light from the lumen 100walls to be collected and imaged within the capsule.

As shown in FIG. 2A, panoramic capsule system 202 includes illuminatingsystem 12 and a camera that includes optical system 14 and image sensor16. A semiconductor nonvolatile archival memory 20 may be provided toallow the images to be retrieved at a docking station outside the body,after the capsule is recovered. Panoramic capsule system 202 includesbattery power supply 24 and an output port 26. Panoramic capsule system202 may be propelled through the GI tract by peristalsis.

Illuminating system 12 may be implemented by LEDs. In FIG. 1, the LEDsare located adjacent to the camera's aperture, although otherconfigurations are possible. The light source may also be provided, forexample, behind the aperture. Other light sources, such as laser diodes,may also be used. Alternatively, white light sources or a combination oftwo or more narrow-wavelength-band sources may also be used. White LEDsare available that may include a blue LED or a violet LED, along withphosphorescent materials that are excited by the LED light to emit lightat longer wavelengths. The portion of capsule housing 10 that allowslight to pass through may be made from bio-compatible glass or polymer.

Optical system 14, which may include multiple refractive, diffractive,or reflective lens elements, provides an image of the lumen walls onimage sensor 16. Image sensor 16 may be provided by charged-coupleddevices (CCD) or complementary metal-oxide-semiconductor (CMOS) typedevices that convert the received light intensities into correspondingelectrical signals. Image sensor 16 may have a monochromatic response orinclude a color filter array such that a color image may be captured(e.g. using the RGB or CYM representations). The analog signals fromimage sensor 16 are preferably converted into digital form to allowprocessing in digital form. Such conversion may be accomplished using ananalog-to-digital (A/D) converter, which may be provided inside thesensor (as in the current case), or in another portion inside capsulehousing 10. The A/D unit may be provided between image sensor 16 and therest of the system. LEDs in illuminating system 12 are synchronized withthe operations of image sensor 16. One function of control module 22 isto control the LEDs during image capture operation.

FIG. 2B shows an alternative swallowable capsule system 210. Capsulesystem 210 may be constructed substantially the same as panoramiccapsule system 202 of FIG. 2A, except that archival memory system 20 andoutput port 26 are no longer required. Capsule system 210 also includescommunication protocol encoder 220, transmitter 226 and antenna 228 thatare used in the wireless transmission to transmit captured images to areceiving device attached or carried by the person being administeredwith a capsule system 210. The elements of panoramic capsule system 202and panoramic capsule system 210 that are substantially the same aretherefore provided the same reference numerals. Their constructions andfunctions are therefore not described here repeatedly. Communicationprotocol encoder 220 may be implemented in software that runs on a DSPor a CPU, in hardware, or a combination of software and hardware.Transmitter 226 and antenna system 228 are used for transmitting thecaptured digital image.

The panoramic camera systems shown in FIG. 2A and FIG. 2B are based on asystem using a pyramidal reflective element having multiple reflectiveside facets facing in different directions. Each of the reflective sidefacets is associated with a component image in its respective direction.The panoramic camera system combines the component images to form acomposite image. There are also other types of panoramic camera systems.For example, in U.S. patent application Ser. No. 11/642,275, entitled“In vivo sensor with panoramic camera” and filed on Dec. 19, 2006, apanoramic camera system using a panoramic annular lens is described. Thepanoramic annular lens is configured to enable images to be captured bythe panoramic camera radially about the longitudinal axis and onto thesingle image plane. The panoramic images captured by the system havingpanoramic annular lens reflective side facets represent continuous fieldof view up to 360°. On the other hand, the panoramic images captured bythe system having a reflective element with multiple reflective sidefacets may represent multiple contiguous fields-of-view.

Though the panoramic images may correspond to a 360° view of the lumen,a practical and convenient way to view the panoramic images is on adisplay screen which is essentially flat. Therefore the panoramic imagehas to be properly placed on the flat screen for viewing. For example,the panoramic image captured by the panoramic camera system with a4-sided reflective element has 4 component images. Each component imagecorresponds to an image captured in a perspective direction and eachcomponent image may be slightly overlapped with its two neighboringcomponent images. The 4 component images are connected in a circularfashion. Images captured by a panoramic camera having a panoramicannular lens will provide continuous fields-of-view and have no boarderlines within the image.

FIG. 3A shows a display screen 300 for displaying the sequence ofcapsule images. The display screen 300 may reserve some screen areas 320and 330 for displaying other information such as patient informationassociated with the underlying image sequence and/or the locationinformation, if available, of the current image being shown. The area310 is designated as the display window for showing the sequence ofcapsule images. The image size from capsule cameras usually is muchsmaller the display screen size. The display window is capable ofproviding multiple viewing windows to accommodate multiple sub-sequencesof image sequence concurrently without sacrificing image resolution.FIG. 3B illustrates an exemplary multiple viewing window configurationwhere the display contains 8 viewing windows 352-368 (arranged as 2rows×4 columns) for concurrent display of 8 sub-sequences of images. The8 sub-sequences are derived from the original sequence and thecollection of the sub-sequences is called aggregated video. Each viewingwindow should be used to display one sub-sequence. However, a user maydecide not to use all the viewing windows at his/her choice. There arevarious possible configurations for multiple viewing windows. Dependingon the size of display window and the size of image, multiple viewingwindows may be configured as 2×2, 2×3, 2×4, 2×6, 3×3, 3×4, 3×6, and etc.The viewing windows may be connected to each other without any space inbetween, or having some space in between as shown in FIG. 3B. The imagesub-sequence for each viewing window may be determined independently.For example, the image corresponding to a sub-sequence in a viewingwindow may be flipped horizontally while the image corresponding toanother sub-sequence in another viewing window may be flippedvertically. The images also may be subject to some pre-processing beforethey are displayed in the view window. For example, the image may betrimmed to remove portions around the boarders, enhanced to improve thevisibility, or coordinate transformed to correct geometric distortion.When the multiple sub-sequences are displayed in the multiple viewingwindows concurrently, it may require more cognitive efforts to identifyanomaly in the multiple sub-sequence. It may be desirable to display themultiple sub-sequences at a slower frame rate than to display a singlesequence. For example, while a single sequence is typically displayed at30 frames per second, the 8 sub-sequences may be displayed at slowerframe rate, such as 20, 15 or other frame rate. On the other hand, ifthere is no movement or little movement for a period of time, it isdesirable to display the aggregated video at faster frame rate duringthis period.

For images from a panoramic capsule system, the configuration formultiple viewing windows may be different. For example, a panoramicimage captured from a panoramic camera system having a 4-sidedreflective element can be shown as a single image 410 in FIG. 4A. Thestructure of the 4-sided reflective element will result in a border linebetween 2 neighboring component images. The 4 component images arelabeled as W 412, N 414, E 416 and S 418 corresponding to 4 directionsof the 4 reflective sides. The panoramic image 410 shown in FIG. 4A iscreated by disjoining the component images corresponding to the Wdirection and the S direction. The resulting panoramic image 410contains component images W, N, E and S from left to right. Note thatwhile the component images are labeled with W, N, E and S directions,these 4 directions are relative directions and any of the 4 componentimages can be designated as the N-direction component image. Similarly,the panoramic image may be disjoined at any other boarder, such as theboarder between the N direction and the E direction resulting in apanoramic image containing component images E, S, W, and N from left toright. For a panoramic system using a panoramic annular lens, the imagewill look seamlessly providing continuous field-of-view. The 360-degreepanoramic image can be disjoined at any desired location. The 4component images in the 4-side reflective element camera could bestitched seamlessly by image processing technology and the imageproduced could also be disjoined at any desired location.

The panoramic image may also be displayed by placing component image inits respective direction. For example, the 4 component images arearranged in 4 directions with its orientation rotated to match itsperspective view, as shown in FIG. 4B. The component image 414 is placedin the north position without rotation. The component image 412 isrotated 90 degrees counterclockwise and placed in the west positionwhile the image 416 is rotated 90 degrees clockwise and placed in theeast position. The component image 418 is rotated 180 degrees and placedin the south position. At the center 425 of the 4 component imagesrepresents a virtual location corresponding to the panoramic camerainside the GI track. The 4 component images represent what the panoramiccamera would see in the 4 directions. Again, the W, N, E and Sdirections are relative directions and any component image can bedesignated as the N-direction component image. As mentioned earlier, thepanoramic images captured by a panoramic camera system having apanoramic annular lens do not have the boarder lines. However, suchpanoramic image still can use the same arrangement as shown in FIG. 4B.For the arrangement of FIG. 4B, the panoramic image may be divided into4 sub-images, rotated and placed in respective positions.

One of the main purposes to display the sequence of capsule images isfor diagnostician to analyze and examine the video to spot any possibleanomaly. The factors to take into consideration for determining displayarrangement include a set up for comfortable viewing and less eyefatigue, and efficient viewing time. For both traditionally colonoscopyand capsule colon endoscopy, the fatigue factors become a major problemin efficacy. With the rampant colon cancer rate, all population above40-50 years old are recommended for regular colon examination, but thereare only limited doctors. For traditional colonoscopy the detection ratedrops after 3-5 procedures because the procedure requires about 30minutes of highly technical maneuver of colonoscope. For capsule colonendoscope each reading of 10's or 100's of images per patient couldeasily make doctors fatigued and lower the detection rate. The vastmajority public do not comply the recommendation for regular colon checkup due to the invasiveness of the procedure. The capsule colon endoscopeis supposed to increase the compliance rate tremendously. Consequently,the issue of reducing fatigue is critical in order to serve theincreased number of colonoscopy procedures. The other critical issue iscost. The doctor's time is expensive and is the major component amongboth colonoscopy procedures. If the viewing throughput rate can beincreased, the total healthcare cost will be substantially reduced.Currently the waiting time for a colonoscopy examination appointment isabout several weeks, or may even be several months. With the dramaticincrease in compliance rate helped with the use of capsule endoscope,there may not be enough doctors to meet the increasing demand.Therefore, methods and systems to reduce the viewing time withoutcompromising the detection rate has another important meaning. Thepanoramic image shown in FIG. 4A is an intuitive arrangement. However,based on actual viewing experience, the image having extremely largeaspect ratio (the ratio of picture width to picture height) may oftencause eye fatigue. By placing multiple panoramic images in the samedisplay window as shown in FIG. 5 and FIG. 6, it reveals a surprisinglypleasant viewing experience. In both FIG. 5 and FIG. 6, the composedimages have the same picture width while the total picture height isincreased. Such arrangement effectively changes the picture aspect ratioto a lower value. The aspect ratios for FIGS. 5A-C and FIGS. 6A-B areclose to that of cinema viewing. Furthermore, more images are displayedin the display window of FIGS. 5A-C and FIGS. 6A-B, which implies that ashorter viewing time is required if the video is played back at the samepicture rate as before.

The single image strip with high aspect ratio will not only causefatigue but also will slow down the video reading speed. When a viewerviews the video, the natural inertia is to focus on the middle and thenlook at one side, and then the other side. If some parts of the video onthe left end attract viewer's attention, the viewer still needs to lookat the right end later. This dynamic tends to slow down the videoviewing and the continuous and strenuous eyeball movement will quicklyget the viewer fatigued.

In FIG. 5A, two panoramic images 510 and 512 are displayed on screen atthe same time. These two panoramic images 510 and 512 are selected froma sequence of panoramic images for viewing. When images correspond to animage sequence are displayed sequentially at a certain frame rate(number of frames per second), the images render themselves as a video.The display locations of the two panoramic images actually define twovideo viewing windows. Each of the two windows 510 and 512 can be usedto display a sequence of panoramic images. Methods to create multiplesub-sequences based on a received sequence of panoramic images will bepresented later. The multiple sub-sequences consist of multiple membersequences which are derived from the original sequence. While the twopanoramic images shown in FIG. 5A have the same up-right orientation,one of the two images may be displayed upside-down, i.e., being flippedvertically as shown in FIG. 5B, where the panoramic image 514 isvertically flipped image 512. Alternatively the image 512 may stay inthe same orientation and the image 510 is flipped vertically. When acapsule camera travels through the GI track, the captured images willappear to move mainly in one direction when the images are shownsequentially on a screen. The configuration of FIG. 5A contains twovideo windows having images in the same orientation. When the two membersequences are played back as video, the contents in the two membersequences will appear to move in the same direction. On the other hand,the contents in the two member sequences corresponding to FIG. 5B willappear to move inward or outward from the center of the two videodisplay windows. It can be a viewer's personal preference to view thetwo member sequences moving in the same direction or movinginward/outward from the center.

FIG. 5C shows an alternative arrangement for displaying multiplesub-sequences where the second member sequence consists of a shiftedversion of the images from the original sequence. As shown in FIG. 5C,an object happens to be located between the component images W and S.Since the panoramic image is formed by stitching, from left to right,images W, N, E and S. Therefore, the object is shown as two parts 511 aand 511 b at both ends of the panoramic image. This split image makes ithard for the diagnostician to perform the examination. Now, the same setof component images are displayed at the bottom of the originalpanoramic image by shifting the component images 2 positions to theright to form an image of the second member sequence 516, as shown inFIG. 5C. The two parts of the split image are now joined as a completeimage 511 in the member sequence 516. On the other hand, an object 513in the original sequence 510 may now become split into two parts 513 aand 513 b in member sequence 516 if it happens to be located betweencomponent images N and E. The above example discloses the membersequence 516 is a spatially shifted version of the original sequence510. Consequently the images in member sequence 510 will appear inmember sequence 516 in a spatially shifted fashion. In this arrangement,the member sequence 510 will have to contain all the images in theoriginal sequence to ensure every image is displayed.

A first member sequence and a second member sequence may be derived froman original sequence using 2:1 temporal sub-sampling. Since neighboringimages usually have high similarity, the above spatial shifting may beapplied to the second member sequence which is a temporal subset of theoriginal sequence. In this arrangement, the total number of images inthe two member sequences is the same as that of the original sequence.Since two display windows are used and the display time will be reducedto half if the display frame rate maintains the same. In addition, sucharrangement provides a convenient view experience since non-splitobjects are always viewable in the center of the display.

In the case that the member sequence corresponding to image 510 is theoriginal sequence, the second member sequence corresponding to image 516as shown in FIG. 5C will have the same number of images as the originalsequence. If the multiple sub-sequences are displayed at normal speed,it will result in the same amount of viewing time. Nevertheless, thearrangement shown in FIG. 5C still provides several advantages. First,it takes care of the split object issue. An object located between anytwo component images will be always shown properly in one of the membersequences. Another advantage is that a diagnostician may now focus onthe left half, the right half or the center part of the aggregated videowithout missing any component image. For example, the 4 component imageson the left half of the screen include images W, N, E and S which are acomplete set of component images. The 4 component images in the middleinclude images N, E, S and W which again are a complete set of componentimages. Similarly, the 4 component images on the right half of thescreen contain a full set of component images. Therefore, thediagnostician doesn't have to scan images side to side and this willmake the viewing experience much more pleasant and relaxed. While theexample in FIG. 5C shows a panorama image having 4 component images, thepresent invention is also applicable to continuous panorama imageswithout any border within the image. The panorama image is consideredcontinuous by wrapping around the two ends that connect the scene.Therefore, the panorama image is cyclically shifted by half of the imagewidth to generate the second member sequence.

Depending on the layout of the display screen and the size of thepanoramic image, more than two member sequences may be displayed on thescreen at the same time. For example, FIG. 6A shows three panoramicimages being displayed in the same display window where all three images610, 612, and 614 have the same orientation. FIG. 6B shows a similararrangement having three panoramic images displayed concurrently in thesame display window. However, the image 616 in the middle is avertically inverted version of image 612. Therefore, the images 610 and616 will look like they are joined in the middle between the two imagesand provide the same visual sensation as the images 510 and 514 of FIG.5B. Therefore, the image 610 and the image 616 will appear move awayfrom each other or move in toward each other depending on the imageorientation and camera movement. Similarly, the images 616 and 614 willlook like they are joined in the middle between the two images andprovide the same visual sensation as the images 510 and 514 of FIG. 5B.In another arrangement similar to that in FIG. 6A, the orientation ofthe middle image 612 remains the same and both images 610 and 614 areinverted.

The multiple sub-sequences are derived from the original sequence. Onemethod to generate multiple sub-sequences is to perform spatialprocessing on the original sequence. For example, the arrangement inFIG. 5C illustrates an example of spatial processing by cyclicallyrotating the original image. According to FIG. 5C, one member sequenceconsisting of a cyclically shifted version of the original images isgenerated. The cyclically shifted member sequence contains the sameamount of data as the original sequence. The cyclically shifted membersequence along with the original sequence forms multiple sub-sequences.More cyclically shifted member sequences can be formed by cyclicallyshifting the original image by different amount. For example, threecyclically shifted member sequences can be generated from the originalsequence by cyclically shifting by 1, 2 and 3 component imagesrespectively. Along with the original sequence, the set contains 4member sequences. The 4 member sequences may be displayed by stacking upone member sequence on the top of the other. The order of stacking upmay be selected as individual preference. For example, the originalsequence may be place on the top and the member sequences cyclicallyshifted by 1, 2 and 3 component images may be placed below the originalsequence in order. Alternatively, the original sequence may be placed onthe top; the member sequence cyclically shifted by 2 component images isplaced below the original sequence, followed by the member sequencecyclically shifted by 1 component image and 3 component images. Theabove examples are for illustration purpose to demonstrate alternativesof spatial processing to generate member sequences for multiplesub-sequences. Other spatial processing methods to generate membersequences are also possible. The cyclical shifting method is alsoapplicable to images having continuous scenes without borders. Theamount of cyclically shifting may be arbitrary instead of the unit ofcomponent image. In the above example, one of the four member sequencesis the original sequence and the other three are spatially shiftedversion of the original sequence. Therefore the three spatially shiftedmember sequences have the same number of images as the originalsequences. The resulting multiple sub-sequences will take the sameamount of viewing time if it is displayed at a regular frame rate.Alternatively, four sub-sequences may be generated by 4:1 temporalsub-sampling of the original sequence, where the temporal sub-samplingwill be described in more detail later. One of the sub-sequences can beused as a member sequence directly. The other three member sequences canbe derived from the other three sub-sequences by spatially, cyclicallyshifting the respective sub-sequences at different spatial distances.Such multiple sub-sequences are a result of temporal processing of theoriginal sequence followed by spatial processing.

Other than the spatial processing discussed above, there are alsotemporal methods to generate member sequences. One preferred method totemporally derive multiple sub-sequences is shown in FIG. 7 where 3sub-sequences are generated from the original sequence. The originalsequence has a total of 3n images and each of the resulting sub-sequencecontains n images. In the case that the total number of images in theoriginal sequence is not divisible by 3, the last picture in thesequence may be repeated as needed to make the total number divisible by3. The sub-sequence A contains images A₁, A₂, . . . , A_(i), . . . ,A_(n), where i is the index corresponding to the temporal order that theimage is displayed. Similarly, sub-sequences B and C contain imagesB_(i) and C_(i) respectively having index i corresponding to thetemporal order that the images are displayed. The sub-sequences A, B andC may be used as member sequences directly. Therefore, at each timeinstance, respective images A_(i), B_(i) and C_(i) are displayed on thescreen simultaneously. It is preferred that the images displayed on thescreen simultaneously have the maximum similarities among them so thatit is easier for the eyes to visualize and perceive the contents.Consequently, the set of respective images A_(i), B_(i) and C_(i) arechosen from consecutive images of the original sequence as shown in FIG.7. This method of constructing the sub-sequence is often called temporalsub-sampling if the image sequence is treated as a sequence along thetime domain. While the example in FIG. 7 illustrates the case having 3member sequences, it is understood that dividing the original sequenceinto 3 member sequences is not a limitation of the present invention.The original sequence may be divided into any integer number of membersequences for display concurrently on the screen. If the sequence isdivided into M member sequences and the total number of images in theoriginal sequence is not divisible by M, the last image of the originalsequence may be repeated as needed to make it divisible by M. While thesub-sequences may be used as member sequences directly, further spatialprocessing by cyclically shifting the sub-sequences at different spatialdistances may be used to generate the member sequences.

While FIG. 7 shows temporal sub-sapling as the method for constructingmultiple sub-sequences, other methods can also be used. For example, theoriginal sequence may be equally divided into 3 sections and the firstsection is assigned to the member sequence A, the second section to themember sequence B and the third section to the member sequence C asshown in FIG. 8. One advantage of this method is that, often at certaininstances, the capsule camera may stay relatively stationary in somesections while the capsule camera may travel normally at other sections.Therefore, there will be some instances that images in some videowindows show no motion or very little motion so that a diagnostician mayfocus his/her attention on images in other video windows. While theexample in FIG. 8 illustrates the case that the original sequence isequally divided into 3 sections, the present invention can also beapplied to cases that the original sequence is divided into otherinteger number of sections. As mentioned before, the collection ofsub-sequences is called aggregated video. The aggregated video for thepanoramic images may require more cognitive efforts for a diagnosticianto identify anomaly. Therefore it may be desired to display theaggregated video at slower frame rate as mentioned before. On the otherhand, if there is no movement or little movement for a period of time,it is desirable to display the aggregated video at faster frame rateduring this period.

FIG. 9A shows a flowchart for a system embodying the present invention.At step 910, an image sequence from a capsule camera is received. Theimage sequence is used to generate multiple sub-sequences at step 912.The multiple sub-sequences are then used to form the aggregated video atstep 914. The aggregated video is provided for display at step 916. FIG.9B shows a flowchart for an alternative system embodying the presentinvention, where the images are captured by panoramic camera. Imagepre-processing such as cropping, sub-sampling and enhancement isperformed at step 920. The component images from all sides of thereflective element are stitched together to form a panoramic image atstep 922. For some systems such as the system using a panoramic annularlens, the image captured is in a continuous field of view and there isno need for stitching. Therefore the step of stitching may be skippedfor such systems. The collection of images forms an image sequence andthe sequence is divided into M member sequences according a method instep 912. The member sequences are then composed into an aggregatedvideo in step 914 and the aggregated video is displayed in step 916. Thestitching for image with component images is optional. For membersequence method similar to that described in FIG. 7 the sub-sequencingcould be done in real time without receiving all the images completely.

The invention may be embodied in other specific forms without departingfrom its spirit or essential characteristics. The described examples areto be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

The invention claimed is:
 1. A method for displaying video of capsuleimages from a capsule camera system, the method comprising: acceptingpanorama images captured with the capsule camera system; generating aplurality of video member sequences based on the images; composing anaggregated video comprising the plurality of video member sequences;providing the aggregated video: displaying the aggregated video; andwherein the video member sequences are generated by uniformlyinterleaving the panorama images temporally followed by cyclicallyshifting the interleaved images.
 2. The method of claim 1, wherein thevideo member sequences are generated by dividing the images intotemporally consecutive sections.
 3. The method of claim 1, wherein theaggregated video is provided according to a frame rate specified.
 4. Amethod for displaying video of panorama images, having a long edge and ashort edge, from a capsule camera system, the method comprising:accepting panorama images captured with the capsule camera system;generating a plurality of video member sequences based on the panoramaimages; composing an aggregated video comprising the plurality of videomember sequences; providing the aggregated video: displaying theaggregated video; and wherein the video member sequences are generatedby uniformly interleaving the panorama images temporally followed bycyclically shifting the interleaved images.
 5. The method of claim 4,wherein the capsule camera system having multiple cameras arranged tocapture a panorama view.
 6. The method of claim 4, wherein the capsulecamera system having a single panorama-view camera.
 7. The method ofclaim 4, wherein the video member sequences are generated by dividingthe panorama images into temporally consecutive sections.
 8. The methodof claim 4, wherein the aggregated video is composed by arranging theplurality of the video member sequences along the short edge.
 9. Themethod of claim 8, wherein the long edge is horizontally positioned. 10.The method of claim 8, wherein the long edge is vertically positioned.11. The method of claim 4, wherein the aggregated video is providedaccording to a frame rate specified.
 12. A system for displaying videoof images from a capsule camera system, the system comprising: aninterface module coupled to accept panorama images captured with thecapsule camera system; a first processing module coupled to theinterface module for accessing the images and configured to generate aplurality of video member sequences based on the images; a secondprocessing module coupled to the first processing module for receivingthe plurality of video member sequence and configured to compose anaggregated video comprising the plurality of video member sequences; anoutput interface module coupled to receive and to provide the aggregatedvideo; a display module coupled to the output interface module fordisplaying the aggregated video; and wherein the video member sequencesare generated by uniformly interleaving the panorama images temporallyfollowed by cyclically shifting the interleaved images.
 13. The systemof claim 12, wherein the video member sequences are generated bydividing the images into temporally consecutive sections.
 14. The systemof claim 12, wherein the aggregated video is provided according to aframe rate specified.
 15. A system for displaying video of panoramaimages, having a long edge and a short edge, from a capsule camerasystem, the system comprising: an interface module coupled to acceptpanorama images captured with the capsule camera system; a firstprocessing module coupled to the interface module for accessing thepanorama images and configured to generate a plurality of video membersequences based on the panorama images; a second processing modulecoupled to the first processing module for receiving the plurality ofvideo member sequence and configured to compose an aggregated videocomprising the plurality of video member sequences; an output interfacemodule coupled to receive and to provide the aggregated video; a displaymodule coupled to the output interface module for displaying theaggregated video; and wherein the video member sequences are generatedby uniformly interleaving the panorama images temporally followed bycyclically shifting the interleaved images.
 16. The system of claim 15,wherein the capsule camera system having multiple cameras arranged tocapture a panorama view.
 17. The system of claim 15, wherein the capsulecamera system having a single panorama-view camera.
 18. The system ofclaim 15, wherein the video member sequences are generated by dividingthe panorama images into temporally consecutive sections.
 19. The systemof claim 15, wherein the aggregated video is composed by arranging theplurality of the video member sequences along the short edge.
 20. Thesystem of claim 19, wherein the long edge is horizontally positioned.21. The system of claim 19, wherein the long edge is verticallypositioned.
 22. The system of claim 15, wherein the aggregated video isprovided according to a frame rate specified.
 23. The method of claim15, wherein the aggregated video is further processed by intensitytransformation on a partial image basis.